Self-Supervised Learning of Audio Representations From Audio-Visual Data Using Spatial Alignment

Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar human perception that relates aural information. In this work, we present a method for self-supervised representation learning based on spatial alignment (AVSA), more sophisticated task than (AVC). addition correspondence, AVSA also learns location of acoustic content. Based 360$^\text{o}$ video Ambisonics audio, propose selection objects using object detection, beamforming signal towards detected objects, attempting learn sound they produce. We investigate use features represent input, different formats: Ambisonics, mono, stereo. Experimental results show 10 $\%$ improvement first order ambisonics intensity vector (FOA-IV) in comparison with log-mel spectrogram features; object-oriented crops brings significant performance increases action recognition downstream task. A number audio-only tasks are devised testing effectiveness learnt feature representation, obtaining comparable state-of-the-art methods scene classification ambisonic binaural audio.